feat: enabled saving and evaluation for moderator (#271) #272
+5,916
−1,192
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* api doc * add PUT * add an temp example for websocket * websocket * update readme * Update README.md * update websocket live simulation api doc * [autofix.ci] apply automated fixes * update websocket doc * add api server with websocket as well as a client * fix mypy errors * support stopping the chat * add 404 to the status code * fix mypy issue * update the returned message types * redesign websocket api * update websocket, fix mypy error * add example of using websocket * clean code & change to existing functions for simulation * fix typing mismatch * update doc & mypy type fix * add type check for run_async_server * move example --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* add customizable evaluation dimensions * add docs * fix mypy error & refactor examples * add docs for evaluation dimensions * update docs and examples * add test cases and fix mypy issue * fix mypy issue * Fix test_create_custom_dimension to use CustomEvaluationDimension.get(pk) (#262) Co-authored-by: openhands <[email protected]> * Fix/custom eval dimension test (#263) * Fix test_create_custom_dimension to use CustomEvaluationDimension.get(pk) * Update documentation for SotopiaDimension and EvaluationDimensionBuilder * [autofix.ci] apply automated fixes * Add API documentation for evaluation dimensions * Refine API documentation for evaluation_dimensions.py to match style * [autofix.ci] apply automated fixes --------- Co-authored-by: openhands <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * add doc --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: openhands <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
…lationshio (#265) * temp run * add relationship api * fix mypy error * update relationship api * simulate episode non-streaming * modify sim episodes * add simulation status * task error * add background task * [autofix.ci] apply automated fixes * back to arun one episode * upload the code * use rq to execute background tasks * temp sol --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* initial framework * initial conv * fix module error * feat: Add 3 new features to Moderator (#266) * feat:introduce booting procedure, saving, and ending chat to moderator * fix: moderator will now ignore none AgentAction, Observations now don't have to include all channels in the mapping * merge changes of example into the original one * fix: 1. save() method now accepts push_to_db config 2. booting()'s waiting time is changed to 0.1 sec * fix: rewrite booting() so that different agent will receive different background information * fix: moderator now inherits from Node directly, instead of from BaseAgent --------- Co-authored-by: JXZhou <JXZhou> * add save condition for moderator * push to db false * to fully stop * stopping all agents * fix mypy * fix mypy error --------- Co-authored-by: JXZhou <[email protected]>
* prototype for modal serving * add openai secret * fix type annotation * add doc * bug fix for simulation api * add customize model, evaluator model and evaluation dimensions * Implement modal API server with Redis integration and FastAPI setup - Added a new script for the modal API server that initializes a Redis instance. - Created a persistent volume for Redis data and included a function to download initial data if not present. - Configured a Docker image with necessary dependencies including Redis Stack and FastAPI. - Implemented a web API class that sets up and cleans up the Redis connection, ensuring readiness before serving requests. - Integrated the SotopiaFastAPI application within the modal framework. --------- Co-authored-by: XuhuiZhou <[email protected]>
* initial * initial ui * merge main * add new ui * switch to fastAPI * websocket check * fix render episode error * add page; make a simplified page and still WIP * [autofix.ci] apply automated fixes * fix simplified streaming version * semi-done character page + avatar assets * Fixed character card styling * [autofix.ci] apply automated fixes * unified rendering and chat display * updated chat character icons * add some tags * add typing * temp fix * add characters avatar to simulation * fix episode full avatar * go to modal config * clean up code * add modal streamlit app * clean codebase except websocket * remove repeated local css * clean websocket * fix get name error * fix errors * pre render scenario * add custom eval * change streamlit to dynamic path * new uv * revert to previous install commands * a fix for modal * add customized dimension * [autofix.ci] apply automated fixes * sort scenarios in simulation * for demo video * update deploy instruction * update intro page * update intro page * [autofix.ci] apply automated fixes * update intro page * add customized dimensions * update api link and modal environment * move folder * fix relative import * update modal image build * use uv to build environment * change folder name * change test * fix modal serve * environment change * refactor * fix ui --------- Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: astrophie <[email protected]>
* feat: enable saving AgentProfile and chat history on redis * fix: AgentProfile can now be loaded by EpisodeLog correctly * feat: enable evaluation of EpisodeLog * [autofix.ci] apply automated fixes * fix: use EpisodeLog and AgentProfile from sotopia directly --------- Co-authored-by: JXZhou <JXZhou> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
XuhuiZhou
commented
Jan 7, 2025
| if len(self.message_history) == 1: | ||
| self.message_history[0].append( | ||
| self.message_history.append( | ||
| [ | ||
| ( | ||
| agent_action.agent_name, | ||
| "Environment", |
There was a problem hiding this comment.
@JXZhou0224 let's think this a bit more carefully, a messge should be (sender, receivers (seperated by comma), message content), so maybe we should not put environment here anymore
There was a problem hiding this comment.
Certiainly! I will fix it. I set the receiver to "Enivronment" because in the render_for_human() method in EpisodeLog requires the message to be sent to "Environment" in order to render successfully. I think I will change it to "All" to represent sending to everyone and rewrite the render_for_human() method.
There was a problem hiding this comment.
Ah I see, maybe that's fine if environment defaults to all.
XuhuiZhou
commented
Jan 7, 2025
XuhuiZhou
commented
Jan 8, 2025
| last_turn=self.scenario, | ||
| last_turn=json.dumps( | ||
| { | ||
| "use_pk_value": self.use_pk_value, |
There was a problem hiding this comment.
The hyperparameter that determines "whether to use AgentProfile from exisiting database" is set in the moderator. Therefore, moderator will communicate this information to all Agents before the start of conversation.
There was a problem hiding this comment.
Sounds good! It's okay if we design things that each agent initialize their own background. However, things could be tricky as the agent has no idea of other agents background and the general scenario. Unless we have a second step to make that happen.
Another easier way is to follow the original sotopia implementation:
Can you follow the reset here to write a similar version of initiating the simulation but allows more than two agents?
sotopia/sotopia/envs/parallel.py
Line 186 in d7724db
XuhuiZhou
commented
Jan 8, 2025
| epilog.save() | ||
| # print(epilog.render_for_humans()) | ||
| if self.push_to_db: | ||
| epilog.save() | ||
| return epilog |
There was a problem hiding this comment.
Maybe we need to change this to evaluation queue or something
XuhuiZhou
commented
Jan 8, 2025
| self.model_name: str = model_name | ||
| self.agent_profile_pk: str = agent_pk | ||
| self.name: str = agent_name | ||
| self.background: dict = background |
There was a problem hiding this comment.
Note self.background is not used anywhere when generating the agent actions
|
@JXZhou0224 I have changed the base from demo to main, please Please also run |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
feat: enable saving AgentProfile and chat history on redis
fix: AgentProfile can now be loaded by EpisodeLog correctly
feat: enable evaluation of EpisodeLog
[autofix.ci] apply automated fixes
fix: use EpisodeLog and AgentProfile from sotopia directly
Closes #
📑 Description
✅ Checks
type/descript(e.g.feature/add-llm-agents)ℹ Additional Information